Spherical perspective on learning with normalization layers

Normalization Layers (NLs) are widely used in modern deep-learning architectures. Despite their apparent simplicity, effect on optimization is not yet fully understood. This paper introduces a spherical framework to study the of neural networks with NLs from geometric perspective. Concretely, radial invariance groups parameters, such as filters for convolutional networks, allows translate steps L2 unit hypersphere. formulation and associated interpretation shed new light training dynamics. Firstly, first effective learning rate expression Adam derived. Then demonstration that, presence NLs, performing Stochastic Gradient Descent (SGD) alone actually equivalent variant constrained hypersphere, stems framework. Finally, this analysis outlines phenomena that previous variants act importance process experimentally validated.